This invention relates to a system for effecting synchronization between a video image reproduced by a video reproducing apparatus, such as a video tape player or a video disk player and the computer processing data representing the image, and particularly to a system for producing graphics such as text and figures by use of a computer, synchronizing the produced pictures and the video images within a recording medium, and displaying a composite image.
This system has many applications in various different fields, such as presentation, education, training and video production. For example, in a presentation which is made in meetings, it may be desirable not only to use a conventional overhead projector or slide for the explanation of drawings, but also to use video images and computer graphics for a more impressive presentation. By combining a computer such as a personal computer or work station, a large display and a video disk player, it is possible to produce and display graphics for presentation together with reproduced and controlled video images.
In this system, in order that the video images recorded by a video camera will be more understandable to the viewers, text and graphics for supplementary explanation of the video images are produced by a computer and displayed together with the video images. For example, while video images recorded in a car race are being displayed, the map of the circuit which has been produced by graphics is compounded with the video images on the display, so that it can be known where the currently displayed car is running in the course.
The operator is able to define the contents to be processed in synchronism with the video image by use of a coordinates-input unit such as a keyboard or a mouse, while viewing the display screen. Moreover, the operator is able to draw graphics directly on the video image and display them together with the video image in a composite manner while viewing the display screen.
An video disk has become the object of public attention as a large-capacity image information recording medium. The video disk can store video information in an amount of tens of thousands of pictures. Also, by the use of the video disk player it is possible to reproduce any one part of the video information from the video disk as a still picture and to continuously reproduce a series of images as a motion picture to be displayed. In general, each picture composing a motion picture is called a frame.
Some video disk players can be controlled to reproduce video images using a computer. This video disk player is connected through standard communication means, such as a RS232C part to the computer. The computer transmits a command to the video disk player, thereby making it possible to control and acquire information as follows:
(1) To reproduce any one of the frames from the video disk as a still picture; PA1 (2) To start the reproduction of video images from a specified frame; PA1 (3) To stop the currently reproduced images at an arbitrary time point; PA1 (4) To transmit information from the video disk to the computer when a previously specified frame is reproduced; and PA1 (5) To send a request for information about the currently reproduced frame from the video disk. PA1 (1) In order for the computer to execute the process in synchronism with the video images, it is necessary to develop for each application a program for monitoring the progress of video image reproduction and for invoking a specified process when a specified frame is reproduced. PA1 (2) Since graphics are produced independently of the video image before the graphics are compounded with the video image, it is difficult to produce the graphics associated with the position, movement, color and so on of the objects within the video image. PA1 (3) In methods in which the video image and graphics are previously compounded, it is difficult to interactively change the graphics in accordance with the situation upon reproduction. In addition, as compared with methods in which video images and graphics are previously and separately stored, and upon reproduction, compounded and displayed, many recording media are necessary. PA1 (1) Image selection step: to select at least one of a plurality of images stored in a video disk or the like. PA1 (2) Process definition step: to define a desired process to be executed in synchronism with the display of the image selected at the image selection step. PA1 (3) Process storing step: to store the contents of the process defined at the process definition step. PA1 (4) Process execution process: to execute the process in synchronism with the display of the stored image after the definition of the process at the steps (1) to (3). PA1 (1) Frame selection step: to select at least one of video images which can be reproduced from the video reproducing apparatus. PA1 (2) Frame display step: to display the frame selected at the frame selection step. PA1 (3) Drawing graphics step: to draw graphics on the display on which the frame is being displayed at the frame display step. PA1 (4) Graphics storing step: to store the graphics drawn at the drawing graphics step in association with the frame (called as the graphics-defined frame) displayed while the graphics are being drawn. PA1 (5) Compound display step: to compound and display the graphics stored in association with the frame in synchronism with the later reproduction of the graphics-defined frame.
A system for controlling a video disk player using a computer is described in SIGGRAPH '80 Conference Proceedings, July (1980), pp 32-42. In this system, the operator orders the computer to control the direction (normal direction or opposite direction), speed, start and stop operations of the reproduction and so on the video disk player. However, once the computer commands the video disk player to start reproduction, the video disk player itself continues to reproduce video images until the computer commands the video disk player to stop the reproduction. During the reproduction, the processing in the computer is not synchronized with the reproduction of video images.
In order to execute the computer processing in synchronism with the video images reproduced by the video disk player, it is generally required to produce a computer program which includes the following steps.
Step 1: monitor the video reproduction until a frame to be processed by the computer is reproduced.
Step 2: start a specified process just when the frame has been reproduced.
A typical example of the process to be executed by the computer in synchronism with video images is the display of graphics. The video images recorded by a video camera are compounded and displayed with text, figures and so on which have been produced by use of a computer or the like, as seen in television broadcasting. Particularly, text compounded and displayed with a video image is called a Telop and is frequently used.
So far a Telop, figures and so on have been produced independently of the video images recorded by the camera, and compounded therewith upon broadcasting or recording. In other words, the Telop and figures are not produced while the compounded results are being watched by the operator. After the Telop and figures are completely produced, they are compounded with the video image and examined. Therefore, it is difficult to produce the Telop and figures in association with the position, movement, color and so on of the objects within the video image.
A system, in which the text and graphics produced by the computer can be synchronously compounded and displayed with the video images reproduced from the video disk player, is presented in Communications of the ACM, Volume 32, Number 7, July (1989), pp 802-810. In this system, texts and graphics can be displayed in synchronism with the video image reproduced by the video disk player, but a method for defining the display and synchronization of the text and graphics with the video image, particularly the user interface is not disclosed.
The method of compounding video images with graphics upon reproduction has been discussed above. On the other hand, a method is often used in which compounded video images and graphics are recorded on a single medium (for example, a video disk or a disk of the computer), and is simply reproduced. This method needs no synchronization and compounding processing upon reproduction and thus has an advantage that simple apparatus and processing can be used. Particularly, under the performance of the present computer, it is often impossible to display complicated graphics in synchronism with video image in real time. In this case, it is inevitable to provide the previously compounded video images before reproduction. However, this method cannot change the contents of graphics to be compounded in accordance with the situation upon reproduction. For example, there is a case in which it is desirable to change the contents of graphics to be compounded and displayed in accordance with the age and nationality of viewers. For English and American persons, English notes are compounded and displayed, but for Japanese, Japanese explanations are given. In addition, comic pictures are compounded and displayed for kindergarten students, and large letters are used for old men. In the method in which the previously compounded video images are provided, it is necessary to provide many recording media, or video disks for each case. Even in that case, the video disks cannot be switched swiftly upon reproduction, or it can be said to be a poor adaptation-to-circumstances system.
Accordingly, the prior art has the following drawbacks.